{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 25 - k-Nearest Neighbors classifier 1\n",
    "\n",
    "We will continue using the Titanic training and test data from [Kaggle](https://www.kaggle.com/c/titanic) from Lab 24.\n",
    "\n",
    "First, we need to install a new library, called [scikit-learn](https://scikit-learn.org/stable/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --user sklearn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next import the necessary libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading and cleaning the data\n",
    "\n",
    "Read the CSV file `train.csv` into the dataframe `train`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display your `train` dataframe below to check it was created properly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For reference, let's get the summaries of all quantitative columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the summaries of all qualitative columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As in Lab 24, we can see there is some missing data in the `Age`, `Cabin`, and `Embarked` columns.  With previous datasets, we have simply removed any rows with missing data.  Today we will take a different approach, and replace the missing `Age` and `Embarked` data with the most likely value.  We won't use the `Cabin` column in the classification, so we don't worry about the missing data in it.\n",
    "\n",
    "First, let's replace any missing ages with the median age.  Compute the median age, and store it in the variable `median_age`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To fill the NaN values in the `Age` column, type and run this code below: `train[\"Age\"] = train[\"Age\"].fillna(median_age)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To check this worked, display the quantitative column summaries again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Did the mean age change?  Did the median age change?  Does this make sense?\n",
    "\n",
    "### First prediction\n",
    "\n",
    "Let's use the quantitative columns of age, fare, SibSp, and Parch to make a prediction.\n",
    "\n",
    "First we will make a data frame with only those columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the variable `y` to be the `Survived` column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will split our (training) data into a training and a test set.  We do this to be able to easily check our predications without using Kaggle (which takes some time)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train,X_test,y_train, y_test =train_test_split(X,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the variables `X_train`, `X_test`, `y_train`, `y_test`.  What is stored in them?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will create a variable to store information about our k-nearest neighbors classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "knn = KNeighborsClassifier(n_neighbors=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many neighbors is the classifier using?\n",
    "\n",
    "Next we will *fit* our classifier using the training data.  This means the classifier stores the information about the coordinates of each training data point and whether it is a survivor in a way that will be make the next step easy to compute. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "knn.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we will make a prediction about the testing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = knn.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display `y_pred`.  Is it what you expected?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's compute the accuracy of our predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "knn.score(y_test,y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How accurate was our model?  We can make it more accurate by adding the qualitative information about the passengers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding the qualitative column information\n",
    "\n",
    "Next, we'll replace the missing value in the `Embarked` column with the mode of the column.  First compute and display the mode."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What's the mode?  How is it stored: as a string or as a Series?\n",
    "\n",
    "Since the mode is stored as as a Series, the easiest way to replace the missing values is by directly using `\"S\"` as the parameter for `fillna()`.\n",
    "\n",
    "Replace the missing values in the Embarked column with `S`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check that your code worked by displaying the summary of the qualitative columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optional:  Effect of Sex, Pclass, and Embarkedon survival\n",
    "\n",
    "Let's look at the effect of Sex, Pclass, and Embarked on survival.  Create two new dataframes, one containing only survivors and one containing only people who perished."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a bar chart of sex of the survivors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Who was more likely to survive?\n",
    "\n",
    "Now create a bar chart of the people who did not survive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Who was more likely to not survive?\n",
    "\n",
    "Let's look at passenger class (`Pclass`).  Create a bar chart of the passenger class of the survivors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which class of passengers was most likely to survive?\n",
    "\n",
    "Now create a bar chart of the passenger class of the passengers who did not survive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which class of passengers was most likely to not survive?\n",
    "\n",
    "Finally, create a bar chart of the ports the surviving passengers embarked at and a bar chart of the ports the passengers who did not survive embarked at."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Is there a difference in the distributions of embarkation ports for the surviving and non-surviving passengers?\n",
    "\n",
    "### Creating dummy variables\n",
    "\n",
    "The k-nearest neighbor classifier can only use quantitative data.  However, we saw that sex,  and passenger class (`Pclass`) played a large role in whether someone survived or not, and the port of embarkation also played a (less) role.  To use these qualitative columns, we can convert them into quantitative data using *dummy variables*.  A *dummy* or *indicator* variable is a variable that takes the value 0 or 1 depending on whether that data point is in some category.\n",
    "\n",
    "Run the following code to create dummy variables for these columns and display the new dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "train2 = pd.get_dummies(train, columns = [\"Pclass\",\"Sex\",\"Embarked\"], drop_first = True)\n",
    "train2.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What happened?\n",
    "\n",
    "Each of the qualitative columns was replaced by the one or two dummy variable columns.  For example, `Sex` was replaced by `Sex_male` which contains 1 if the passenger was male and 0 if the passenger was female.\n",
    "\n",
    "Why is there no `Sex_female` column?\n",
    "\n",
    "We are almost ready to do our classification.  We just need to drop all remaining qualitative columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train2.drop(\"Cabin\",axis = 1,inplace = True)\n",
    "train2.drop(\"Name\",axis = 1,inplace = True)\n",
    "train2.drop(\"Ticket\",axis = 1,inplace = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is another way to make our `X` dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X2 = train2.drop(\"Survived\",axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we split our data into training and test sets.  Use `X2` and `y` from above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's run the classifier.  First create it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next fit the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next make a prediction about the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally compute the accuracy of this model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Did the accuracy improve?  What happens if you change the number of neighbors?\n",
    "\n",
    "What happens if you use few columns?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}